Least squares solutions of bilinear equations

In this second recitation, we will address the following topics: • Review of the projection theorem. • Derivation of the projection operator on the range of a matrix, using completion of squares. • Solutions of under and over constrained linear equations. Notice that this recitation note is an alternative discussion of the least-squares problem which is more difficult than the one you can find in the lecture notes. We suggest that you go through it as an exercise that will give you a different view and more insight on the topic. Consider the usual Euclidean space R 2 , an arbitrary point y ∈ R 2 and a line S that goes through the origin. y y satisfies y We know that the point S ∈ S that is closest to y − S ∈ S ⊥. This geometrically intuitive property can be generalized to other inner-product spaces. Besides the explanatory aid that we can get from it, the orthogonality principle is very useful on the solution of general minimum distance or least squares problems. The generalization of the minimum distance problem is achieved through the following the projection theorem. It comprises two parts: the existence and uniqueness and the the orthogonality principle. In the lecture notes, the orthogonality principle is proven and subsequently used to derive explicit solutions to the least squares problem. By deriving those explicit solutions, the first part of the projection theorem (existence and uniqueness) is also solved. Theorem 2.1 (Part 1-Existence and Uniqueness) Let V be a Hilbert (complete 1 inner-product) space and S ⊂ V be a finite dimensional subspace of V. Then, for any given y ∈ V the following problem has a unique solution: S = arg min y y − s (1) s∈S Theorem 2.2 (Part 2-Orthogonality principle) Let V be a inner-product space and S ⊂ V be a subspace of V. Then for any given y ∈ V the following two conditions are equivalent: 1 Complete means that we can use the norm of the space to define converging sequences and the notion of limit. You do not have to worry about this technicality because in this course we will only deal with complete spaces. So, you can just think of a Hilbert Space as an Inner-Product space.